Technical Field
The present disclosure relates to a method, apparatus, and non-transitory computer readable medium to objectively measure the quality of software products.
Description of the Related Art
The “background” description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description which may not otherwise qualify as prior art at the time of filing, are neither expressly or impliedly admitted as prior art against the present disclosure.
Software quality has gained a lot of attention from academia and industry due to its important role in modern-day business success. See E. van Veenendaal and J. McMullan, Achieving software product quality: UTN Publishers, 1997, incorporated herein in its entirety. The quality of software is critical and especially important in real-time systems that may lead to loss of human life if a failure occurs in the system. A lot of research in software engineering is done to assess the quality of software. See M. O. D. a. T. P. Allan Baktoft Jakobsen, “Towards a maturity model for software product evaluations,” in Proceedings of 10th European conference on software cost estimation (ESCOM'99), 1999; R. E. A. Al-Qutaish, Alain, “A maturity model of software product quality,” Journal of Research and Practice in Information Technology, vol. 43, pp. 307-327, 2011; A. Alvaro, E. S. d. Almeida, and S. L. Meira, “Towards a Software Component Certification Framework,” presented at the Proceedings of the Seventh International Conference on Quality Software, 2007; A. April and Coallier, “Trillium: a model for the assessment of telecom software system development and maintenance capability,” in Software Engineering Standards Symposium, 1995. (ISESS'95) ‘Experience and Practice’, Proceedings, Second IEEE International, 1995, pp. 175-183, incorporated herein in their entirety. Capability Maturity Model Integration (CMMI) is a process improvement framework that was defined by the Software Engineering Institute (SEI) of Carnegie Mellon University (CMU). SEI philosophy in their concentration on the process is that the produced product quality is highly affected by the quality of the process that is used to develop the software product. In other words, CMMI focus is to improve the development process in an organization based on the following premise “the quality of a system or product is highly influenced by the quality of the process used to develop and maintain it”. See F. B. Abreu, M. Goulão, and R. Esteves, “Toward the design quality evaluation of object-oriented software systems,” in Proceedings of the 5th International conference on Software Quality, Austin, Tex., USA, 1995, pp. 44-57, incorporated herein in its entirety. However, previous research has been convincing that dealing with “process quality” is not sufficient to ensure the product quality, hence the assessment of “software product quality” is also needed. See T. Maibaum and A. Wassyng, “A product-focused approach to software certification,” Computer, vol. 41, pp. 91-93, 2008, incorporated herein in its entirety. Moreover CMMI requires processes used by software development organizations such as project planning, resolving issues in the project, and other measures that are used in the project to be documented. The documents are required to assess the maturity level of the organization's processes or to describe the organization's overall performance. See C. P. Team, “CMMI® for Development, Version 1.3,” CMU/SEI-2010-TR-0332010, incorporated herein in its entirety. This leads to the proposal of a framework that enables the assessment of the final software product (code) without depending on the development process of the software. The proposed framework can evaluate the software product quality without requiring a documented process or a software development methodology thereby making the assessment of the software product much easier.
The term quality is non-specific and there is no general consensus on a definition of the word quality. Gillies defines quality as how successfully a product can satisfy people who use it and how successfully it enables them to realize the benefits of using it. See A. C. Gillies. (1997) “Software Quality” (2nd), incorporated herein in its entirety. Pressman stated that quality is “conformance to explicitly stated functional and performance requirements, explicitly documented development standards, and implicit characteristics that are expected of all professionally developed software.” See R. S. Pressman. (2004), “Software engineering: A practitioner's approach,” incorporated herein in its entirety.
ISO defines quality as “the totality of characteristics of an entity that bear on its ability to satisfy stated and implied needs”. See I.I.I. technology. (1999) “Software product evaluation—Part 1,” incorporated herein in its entirety. On the hand IEEE defines quality as “the degree to which a system, component, or process meets specified requirements and customer (user) needs (expectations)”. See IEEE, “Standard Glossary of Software Engineering Terminology. New York, N.Y., USA, Institute of Electrical and Electronics Engineers,” 1990, incorporated herein in its entirety.
Hence, there are many definitions of quality in terms of both technical and non-technical properties from the user perspective like availability, usability, and maintainability, (i.e. whether the user requirements and needs satisfied). Generally, quality can be assessed from two main points of view: the first is technical and the other is user-oriented. Process and product assessments focus on technical aspects. These days rigor has increased in the development process to produce a reusable, maintainable and robust final product this being the software engineers' view. On the other hand a user oriented perspective concentrates on user satisfaction with the product.
Garvin provided a more complex definition of quality. Garvin defines quality from five perspectives in different domains and categorizes them in two groups as outlined below. See D. A. Garvin, What Does “Product Quality” Really Mean?: Sloan Management Review, 4, 1984, incorporated herein in its entirety.
A. Technical Views                1. Transcendent approach: there is no specific definition for quality. In other words, the term quality can be gained thorough experience not through specific definition.        2. Product-based approach defines quality as a quantitative variable that can be measured through a certain set of attributes that are possessed by the product.        3. Manufacturing-based approach: define quality as how the final product is conformed to the requirements. This approach is concerned on the engineering and manufacturing side.        
B. People-Oriented View                1. User-based approach defines the quality as how the product can satisfy the of the consumers? This approach is the most approach that has a subjective definition of the quality than other approaches.        2. Value-based approach: they define quality in terms of prices and cost of the performance of the product that the customer can afford.        
The manufacturing-based view is the approach that is most popular with software engineers and lies under waterfall development methodology. The two approaches of the people-oriented category are more common than other approaches.
These days academia and industry pay much attention to software quality because they play important roles in modern-day business, and to some extent modern-day living. In the last decade software quality has improved significantly because new techniques and technologies have been adopted to improve software product quality.
Software development companies are recognizing that managing the development process will lead to the production of better quality software. See I. Sommerville, Software Engineering. UK: Addison-Wesley, 2007; M. S. a. M. Niazi, “Systematic review of organizational motivations for adopting CMM-based SPT,” Information and Software Technology Journal, vol. 50, pp. 605-620, 2008; B. Pitterman, “Telcordia Technologies: The journey to high maturity,” IEEE Software, vol. 17, pp. 89-96, 2000; G. Yamamura, “Software process satisfied employees,” IEEE Software, pp. 83-85, 1999, incorporated herein in their entirety. Software process improvement (SPI) approach is widely used to address the effective management of the development process. Research has shown that organizations using SPI in their software development process will lead to production of high quality products, decrease development time and cost, and increase the development productivity. See N. Ashrafi, “The impact of software process improvement on quality: in theory and practice,” Information & Management, vol. 40, pp. 677-690, 2003; G. K. James J. Jianga, Hsin-Ginn Hwang. Jack Huang, and Shin-Yuan Hung, “An exploration of the relationship between software development process maturity and project performance,” Information & Management, pp. 279-288, 2004, incorporated herein in their entirety. The following studies describe the impact of using SPI on software quality. Staples and Niazi performed a systematic literature review in order to investigate the reasons organizations had for adopting CMM-based SPI approaches. The results showed that 70% of the organizations had adopted CMM-based SPI approaches to improve the quality of their products. Yamamura made a survey of an organization that had been assessed as SW-CMM level 5 before and after applying the process improvement. The result showed that there is a correlation between process improvement and satisfaction of employees. Employee satisfaction increased by 26% and average satisfaction moved from “neutral” to “very satisfied” rating. Diaz and Sligo showed that the defect rate decreased by half as the CMM level increased, also defect injection in projects at level 2 was eight times higher than projects at level 5. J. Herbsleb and D. Goldenson showed that increases in the maturity level led to greater success in meeting schedule and budget goals, increased staff morale and customer satisfaction. See M. D. a. J. Sligo. (1997), How Software Process Improvement Helped Motorola. 14, 75-81; J. H. a. D. Goldenson, “A systematic survey of CMM experience and results,” presented at the 18th international conference on software engineering (ICSE-18), Berlin, Germany, 1996, incorporated herein in their entirety. For example the ability to meet schedules and the budgets increased from 40% for companies at CMMI level 1 to 80% and 78% for companies at CMMI level 3. Customer satisfaction increased from 80% for companies at CMMI level 1 to 100% for companies at CMMI level 3.
Nowadays software quality plays an important role in developing different types of software applications. The quality of software is critical and important, especially in real-time systems that may lead to loss of human life when a failure occurs in the system. There is no general consensus on the definition of software quality. A model can be defined as an abstract view of the reality that eliminates the details. There are different types of models such as the cost estimation model, maturity model, quality model, etc. Quality models help software engineers, developers, project managers, software customer, etc. in assessing the quality of the software product to decide whether it meets their requirements or not. Also software organizations use quality models to evaluate their final product to decide if the product is ready for deployment. Software metrics are used to assess the desired related quality attributes in the quality models. Usually software quality models are organized in multi-levels. The highest level contains the software quality attributes (called quality characteristics or factors) like: maintainability, testability, etc. External quality attributes comprise a set of internal quality attributes (called sub-characteristics) which depend on them such as: coupling, cohesion, etc.
There are several quality models in the literature for different types of applications such as desktop, mobile, components, web-services and web applications. The focus here will be on popular software quality models.
McCall et al defined the software product quality model to guide acquisition managers in U.S. Air Force Electronic Systems Divisions and Rome Air Development Centers. See J. A. McCall, P. K. Richards, and G. F. Walters, “Factors in software quality. Volume I. Concepts and Definitions of Software Quality,” DTIC Document 1977, incorporated herein in its entirety. McCall's model is one of the most well-known quality models in software engineering. It consists of a hierarchy of three levels which are: factors, criteria, and metrics; 11 quality factors (external quality attributes) which reflect the user's view; and 23 quality criteria (internal quality attributes) which reflect the developer's view as shown in Table 1.1.
TABLE 1.1McCall's Quality Model Factors and CriteriaFactorsCriteriaCorrectnessTraceabilityConsistencyCompletenessReliabilityError toleranceSimplicityConsistencyAccuracyEfficiencyExecution efficiencyStorage efficiencyIntegrityAccess controlAccess auditUsabilityOperabilityCommunicativenessTrainingMaintainabilitySimplicityConcisenessConsistencySelf-descriptivenessModularityFlexibilitySelf-descriptivenessExpandabilityGenerality
Each Factor in McCall's quality model can be defined as follows:                1. Correctness: The extent that the program can fulfill the user requirements.        2. Reliability: The ability to perform the job preciously as required.        3. Efficiency: The amount of resources required to perform a task.        4. Integrity: The extent to which the program can protect itself from unauthorized activities.        5. Usability: The required effort from the user in order to use the program without difficulties.        6. Maintainability: The ease of finding and fixing bugs in the program.        7. Flexibility: The required effort needed to modify the program.        8. Testability: The ease of testing the program to make sure it meets the requirements.        9. Portability: The effort required to move the program from one environment to another.        10. Reusability: The ease of using the software in other applications.        11. Interoperability: The required effort to link two systems.        
Bohem et al. developed his quality model in 1971 to automate and quantitate the evaluation of the software product. See B. W. Boehm, J. R. Brown, and M. Lipow, “Quantitative evaluation of software quality,” in Proceedings of the 2nd international conference on Software engineering, 1976, pp. 592-605, incorporated herein in its entirety. Bohem's model consists of 3 levels: high-level characteristics, intermediate level characteristics, and primitive characteristics. Intermediate level characteristics are an essential element for high-level characteristics, and primitive characteristics are a necessary condition for intermediate level characteristics from the author's point of view. Bohem's model is considered to be one of the first proposed quality models in software engineering. The higher levels and their lower associated levels are shown in Table 2.
TABLE 2Bohem's Quality ModelHigh-level Intermediate-level PrimitivecharacteristicscharacteristicsCharacteristicsPortabilityDevice-IndependenceSelf-ContainednessAs-Is-UtilityReliabilityAccuracySelf-ContainednessCompletenessRobustness/IntegrityConsistencyEfficiencyAccountabilityDevice EfficiencyAccessibilityHuman Robustness/IntegrityEngineering AccessibilityCommunicativenessMaintainabilityTestabilityAccountabilityAccessibilityCommunicativenessSelf-DescriptivenessStructurednessUnder-ConsistencystandabilitySelf-DescriptivenessStructurednessConcisenessLegibilityModifiabilityStructurednessAugmentability
The quality attributes in the intermediate-level characteristics of Bohem's quality model can be defined as follows:                1. Reliability: The extent to which the program can perform its intended job accurately.        2. Efficiency: The extent to which the program can operate without wasting resources.        3. Human Engineering: The extent to which the program can be used easily by the user.        4. Testability: The extent to which the program meets the requirements.        5. Understandability: The ease of reading the code and clearly comprehending its purpose.        
Dromey developed a software product quality model. Dromey's model recognizes that the quality evaluation of each product is different. See R. G. Dromey, “A model for software product quality,” Software Engineering, IEEE Transactions on, vol. 21, pp. 146-162, 1995, incorporated herein in its entirety. Dromey linked the product properties with the quality attributes in standard ISO-9126. As shown FIG. 1, there are four categories of product properties and seven quality attributes.
The International Organization for Standardization and International Electrotechnical Commission published the first version of ISO/IEC 9126 standard in 1991 which is an international standard for evaluation of software product quality. After that, ISO and IEC expanded ISO/IEC 9126 into the following parts:                1. ISO/IEC IS 9126-1: quality model        2. ISO/IEC TR 9126-2: external metrics        3. ISO/IEC TR 9126-3: internal metrics        4. ISO/IEC TR 9126-4: quality in use metrics        
This quality model contains 6 characteristics which are divided into 27 sub-characteristics for internal and external quality attributes and 4 quality-In-Use characteristics as shown in FIG. 2 and FIG. 3 respectively.
We can measure the characteristic by using the metrics of the sub-characteristics and using an appropriate aggregation method to combine the values of the measurements into a single value. The developer, evaluator, or the quality manager can modify the metrics or add metrics which are not listed in the quality models.
The external/internal quality attributes can be defined as the following:                1. Functionality: The extent to which the software functionality meets the requirements under the specified conditions.        2. Reliability: The ability of the software to operate under the specified conditions.        3. Usability: The ease of learning and using the system.        32        4. Efficiency: The extent to which the software can operate with performance that is relative to the resources that are used.        5. Maintainability: The ease of modifying the software, fixing bugs, and adding additional features.        6. Portability: The ability of the software to be moved from one environment to another.The quality-in-use quality attributes can be defined as the following:        1. Effectiveness: “The capability of the software product to enable users to achieve specified goals with accuracy and completeness in a specified context of use.”        2. Productivity: “The capability of the software product to enable users to expend appropriate amounts of resources in relation to the effectiveness achieved in a specified context of use.”        3. Safety: “The capability of the software product to achieve acceptable levels of risk of harm to people, business, software, property or the environment in a specified context of use.”        4. Satisfaction: “The capability of the software product to satisfy users in a specified context of use.”        
Samoladas et al. defined a quality model to evaluate open source systems. The new quality model called Software Quality Observatory for Open Source Software (SQO-OSS). The SQO-OSS model is constructed using GQM approach and ISO-9126 model. The authors provide metrics to measure the quality attributes of the model based on its definition on ISO-9126. SQO-OSS model is shown in FIG. 4. See I. O. f S. (ISO), “ISO/IEC IS 9126: Software Product Evaluation—Quality Characteristics and Guidelines for their Use,” 1991; I Samoladas, G. Gousios, D. Spinellis, and I. Stamelos, “The SQO-OSS quality model: measurement based open source software evaluation,” in Open source development, communities and quality, ed: Springer, 2008, pp. 237-248; V. R. Basili, G. Caldiera, and H. D. Rombach, “The Goal Question Metric Approach, Chapter in Encyclopedia of Software Engineering,” ed: Wiley, 1994, incorporated herein in their entirety.
Olsina et al reused and extended ISO 9126-1 quality model to develop a quality model for web application. See L. Olsina, R. Sassano, and L. Mich, “Towards the Quality of Web 2.0 Applications,” in 8th International Workshop on Web Oriented Software Technology (IWWOST2009), 2009, incorporated herein in its entirety. The author added a “Quality Content” characteristic to ensure the quality of information (text) in the website. “Quality Content” can be defined as “the capability of a Web product to deliver information which meets stated and implied needs when used under specified conditions”. The new characteristic is divided into 4 sub-characteristics. The extended model is shown in FIG. 5.
Franke et al. proposed a software quality model for mobile application. See D. Franke, S. Kowalewski, and C. Weise, “A mobile software quality model,” in Quality Software (ASIC), 2012 12th International Conference on, 2012, pp. 154-157, incorporated herein in its entirety. The author used McCall's, Bohem's, and ISO-9126 quality models in developing his model. The authors included only the most important quality attributes for mobile software so it would be easier for the developers to focus on these quality attributes. The proposed model can be easily extended for a specific application. The proposed model contains 7 characteristics as shown in FIG. 6.
The quality attributes of Franke's model can be defined as the following:                1. Flexibility: The effort required to modify the software so as to be able to work on it in another environment.        2. Extensibility: The effort required to extend the software.        3. Adaptability: The extent to which the software can be adapted to changes.        4. Portability: The effort required to make the software runs on a different platform.        5. Usability: The ease of using the software for an end user.        6. Efficiency: The amount of resources used by the software to do the work.        7. Data Persistence: The capability of the software to save its state.        
Zahra et al. proposed a quality model for mobile application. See S. Zahra, A. Khalid, and A. Javed, “An Efficient and Effective New Generation Objective Quality Model for Mobile Applications,” International Journal of Modern Education and Computer Science (IJMECS), vol. 5, p. 36, 2013, incorporated herein in its entirety. The proposed model is derived from ISO-9126 characteristics and sub-characteristics that can be applied to mobile applications. Zahra's quality model is a general model where the developers can tailor the model for a specific application. The model contains 6 characteristics and 4 sub-characteristics as shown in FIG. 7.
The characteristics of Zahra's model can be defined as follows:                1. Functionality: The extent to which the software meets the requirements and the specification of the user.        2. Usability: The ease of using the software and having an interactive interface.        3. Efficiency: The amount of resources that the software uses as well as the time it takes to do a specific task.        4. Maintainability: The effort required to add extra functionality to the software and the ability of the software to adapt to environment changes.        5. Data Integrity: The ability of the software to save the information in case of crash or pausing the software.        6. Portability: The extent to which the software can run on different platforms.        
Efforts have been made for several decades to improve the quality of software. Software organizations have long been concerned about the quality of their products. See M. Lepmets, “Which Process Model Practices Support Project Success?,” presented at the 17th European Conference, EuroSPI 2010, 2010; M. N. Mark Staples, Ross Jeffery, Alan Abrahams, and a. R. M. Paul Byatt, “An exploratory study of why organizations do not adopt CMMI,” The Journal of Systems and Software, vol. 80, pp. 883-895, 2007; S. Zahran. (1999), incorporated herein in their entirety. Software process improvement: practical guidelines for business success. Many software organizations place customer satisfaction as their highest priority in order to compete with other quality software. See C. V. W. Mark C. Paulk, Bill Curtis, and Mary Beth Chrissis. (1994). The Capability Maturity Model: Guidelines for Improving the Software Process. California, US, incorporated herein in its entirety. Software quality becomes more important as it is increasingly depended upon to run our day-to-day lives. Software process improvement is the approach that is most commonly used of several approaches that have been developed to address software quality issues. See SEI. (2008). Process Maturity Profiles, incorporated herein in its entirety.
SPI offers a powerful way for software organizations to measure their capabilities to develop systems and for them to discover their strong and weak points. Organizations, after identifying their strengths and weaknesses, are able to start process improvement programs and to clearly define achievable goals. SPI helps software organizations understand their development process. These organizations can then identify areas that can be controlled in order to achieve a specific product outcome. See W. S. Humphrey. (1995). A Discipline for Software Engineering, incorporated herein in its entirety.
Rico defined the SPI as “the discipline of characterizing, defining, measuring, and improving software management and engineering processes, leading to successful software engineering management, higher product quality, greater product innovation, faster cycle times, and lower development costs, simultaneously.” See D. F. Rico. (1997, 6-10-2014). Software Process Improvement (Impacting the Bottom Line by using Powerful “Solutions”), incorporated herein in its entirety. O'Regan defined SPI as “A program of activities designed to improve the performance and maturity of the organization's software processes and the results of such a program.” See G. O'Regan. (2010). Introduction to software process improvement, incorporated herein in its entirety. Sommerville stated that “SPI involves understanding existing processes and changing these processes to improve product quality and/or reduce costs and development time”. Process improvement does not mean to apply a specific method or tool that is used by others, rather process improvement should be adopted as an activity peculiar to the organization. Rico and Sommerville focus on the SPI definition to increase the quality of software while decreasing development time and cost. Approaches to Software Process Improvement
There are many studies in software process improvement. We will present the most popular software process improvement approaches in the literature which their focus on the quality of the software.
Capability Maturity Model of Integration (CMMI) is a process improvement framework that was defined by the Software Engineering Institute (SET) of a Carnegie Mellon University (CMU).
CMMI improves the processes performance through providing a set of practices to organizations. CMMI process improvement identifies the strengths and weaknesses in an organization's processes, then it converts weaknesses into strengths by bringing about process changes. See C. Institute. (14-10-2014). CMMI Overview, incorporated herein in its entirety. CMMI has been adopted by over 5000 organizations all over the world. CMMI has a set of best practices that help to improve quality, efficiency, and effectiveness in organizations. The complete list of practices and process area can be found in and respectively. See C. Institute. (14-10-2014). CMMI Overview; C. Institute, “What is CMMI?”; C. Institute. (14-10-2014). CMMI Process Areas, incorporated herein in its entirety
CMMI defines three constellations that can be defined as a collection of best practices and process improvement goals that are used to build models, appraise related documents, and develop training material for an organization's interest area. These goals and practices are grouped into different process areas. The constellations are:                1. CMMI for Development (CMMI-DEV): model is used in process improvement and developing the quality of the services and products in the organizations        2. CMMI for Acquisition (CMMI-ACQ): process model that provides guidance to the organizations for managing the acquisition of services and products that meet customer needs        3. CMMI for Services (CMMI-SVC): model that guides organizations in initiating, managing and deploying the services that meet the customers and end-users requirements        
CMMI for development (CMMI-Dev) v 1.3 is the latest model from the Software Engineering Institute (SEI) that was released in 2010. CMMI is an integration of three source models—Capability Maturity Model for Software (SW-CMM), Electronic Industries Alliance standard 731 (EIA 2002a), and Integrated Product Development Capability Maturity Model (IPD-CMM)—that were selected based on their successful adoption in processes improvement in organizations. CMMI was created to overcome the use of multiple CMMs. The CMMI framework can suit multiple disciplines: software engineering, systems engineering, hardware engineering and Integrated Process and Product Development. See J. Persse, Project Management Success with Seven CMMI Process Areas: Prentice Hall, 2007, incorporated herein in its entirety. It also gives flexibility by supporting two different representations (staged and continuous representations).
CMMI was built using information from well-known models, software development practices of organizations which have a high CMM maturity level and have practiced these models for a long time, and the knowledge of the best systems. FIG. 8 shows the models that were used over time that led to CMMI v 1.3.
The main objective of the CMMI is to measure the degree to which processes are well-defined and managed in organizations, in other words it measures an organization's capability. CMMI describes an evolutionary path to develop and improve the organizations' processes from immature and chaotic processes to mature and optimized processes. The CMMI consists of five maturity levels. These levels are: Initial, Managed, Defined, Quantitatively Managed, and Optimizing (shown in FIG. 9). Maturity level 1 represents the lowest state and the starting point in the staged representation whereas maturity level 5 represents the highest state of maturity that the organization can achieve in the staged representation.
CMMI-Dev contains several process areas (PAs). There are twenty-two PAs in CMMI-Dev that are split as the following: sixteen core PAs that are common to all CMMI models, one shared PA that is shared between at least two CMMI models, and five specific PAs. Each maturity level contains several PAs that should be satisfied to achieve that maturity level, and for each PA there are different goals (both generic and specific) that are described and must be presented to satisfy the PA. For each goal several practices are identified and described in order to help organizations understand how to achieve the goals and the expected results of adopting these practices. For a particular maturity level to be achieved by an organization, all the goals that are associated with the PAs for that level and the lower levels must be satisfied. Applying the practices of higher level maturity without completing all the practices in the lower levels will put at risk their success because the basis of the higher level maturity practices will be incomplete (lower level maturity practices not having been put into practice first).
There are two representations in staged and continuous. In the staged representation, the maturity levels offer an ordered and recommended way of improving the organization processes in stages as shown in FIG. 10. There are PAs within each maturity level and within each PA there are generic and specific goals which contain generic and specific practices. All of these goals and practices in a particular level and lower levels should be maintained to achieve a particular level. There are five maturity levels in the staged representation with the description of each level shown in Table 4.
TABLE 4Maturity Levels of CMMI in Staged RepresentationLevelMaturity LevelDescription1InitialThe processes are chaotic, there is no defined plan. Success mainlydepends on employee experience.2ManagedThe processes are managed, monitored, and controlled according to a dcfincd plan and involve experienced people and stakeholders toproduce the desired output.3DefinedThe processes are well-defined in standards, methods, and tools.These standards are used to maintain consistency across all projects by tailoring standards.4QuantitativelyThe methods used to quantify Managed the quality and performance ofprocesses to manage projects in order to satisfy end-users have been established.5OptimizingThe processes have been continuously improved based onquantitative objectives.
In continuous representation, shown in FIG. 11, specific goals organize specific practices and generic goals organize generic practices and each practice is associated with a capability level. Continuous representation is used by software organizations that are interested in improving a specific process area in the company. There are four capability levels in continuous representation as shown in Table 5 with the description of each level.
TABLE 5Capability Levels of CMMI in Continuous Representation.CapabilityLevelLevelDescription0IncompleteThe process is not performed or not fullyperformed (chaotic process).1PerformedAll the work required to produce the product of thatprocess area is fulfilled. In other words the specificgoals associated with that process area are satisfied.2ManagedThe process is managed according to a well-definedplan that specifies policies, how to execute theprocess, stakeholders, and howto monitor andcontrol the process.3DefinedThe process is tailored from the standardizedprocesses of theorganization.
CMMI is a useful improvement framework with defined levels, PAs, goals, and practices. CMM and its successor CMMI framework does not specify how to execute the practices or provide suggestion on how to implement them thus increasing the flexibility of organizations by allowing freedom to decide how to implement different practices for each process area.
Many organizations want to measure their process improvement progress against the CMMI for the following reasons:                1. To identify areas that can be improved by assessing their processes by CMMI processes        2. To inform clients how successfully they meet CMMI best practices        3. To satisfy customers who have asked them to follow certain CMMI practices before agreeing to establish a working relationship        
This measurement is done by CMMI appraisal. For organizations to use CMMI appraisal they must meet the requirements that can be found in the Appraisal Requirements for CMMI (ARC) document. Standard CMMI Appraisal Method for Process Improvement (SCAMPI) is used for organization appraisal. In other words it is used for conducting the Class A appraisal because it is the only official class. There are three types of appraisals described in the ARC document (Class A, Class B, and Class C). Depending on the requirements of the appraisal, the organization can decide which class is suitable. The appraisal classes with the definition of each class are shown as follows:                1. Class A: Used by the organization after implementing a significant process improvement. It is the only appraisal that utilizes an official method and provides the maturity level in the staged representation or the capability level in the continuous representation. This method is conducted by certified people from the Software Engineer Institute.        2. Class B: This appraisal requires less time and money when compared with the Class A. It is conducted to find the maturity level (unofficial) of the organization where it can be located.        3. Class C: This appraisal is the most flexible, takes less time, and is cheaper than Class A or Class B. It is done by people who are well-trained on CMMI and the organization processes. This method is conducted in order to address the needs that are output from gap analysis.        
SPICE is a set International Standard for Software Process Assessment. See I. I. 15504, “Information technology—Software process assessment,” Type 21998, incorporated herein in its entirety. SPICE provides a reference model that encourages self-assessment of software processes. The reference model describes software engineering processes that include best practices and activities. The purpose of this model is to provide a common basis of different software process assessment models, in such a way that it can help report the results of assessment in a common context even if different process assessment models are used, this will help in comparing between different software process assessment results. The architecture of the reference model has two dimensions: a process dimension and a process capability dimension. It also has nine parts:                Part 1: Concepts and introductory guide        Part 2: A reference model for processes and process capability        Part 3: Performing an assessment        Part 4: Guide to performing assessments        Part 5: An assessment model and indicator guidance        Part 6: Guide to competency of assessors        Part 7: Guide for use in process improvement        Part 8: Guide for use in determining supplier process capability        Part 9: Vocabulary        
The process dimension describes what has to be achieved in order to reach the defined process's purpose using measurements. The process capability dimension is characterized by process attributes which are grouped into six capability levels which have been assigned an ordinal scale. Each higher capability level represents an improvement to the management, performance, and control of the process. Table 6 shows the capability levels and the associated process attributes that can be measured as a percentage scale:
TABLE 6SPICE Capability levels and Process AttributesCapabilityCapability LevelLevelNameProcess AttributesLevel 0Incomplete process—Level 1Performed processProcess performance attributeLevel 2Managed processPerformance management attributeWork product management attributeLevel 3Established processProcess definition attributeProcess resource attributeLevel 4Predictable processMeasurement attributeProcess control attributeLevel 5Optimizing processProcess change attributeContinuous improvement attribute
In order to achieve a specific capability level all the lower capability levels should be completed and all the process attributes of the desired level should be achieved to a certain rate specified by the model for each attribute in the desired level. The process assessment is done by a qualified assessor or by using certain tools for data collection which have been approved by an assessor.
SPICE harmonizes the existing approaches to process improvement but it does not specify how to achieve the process improvements or specify a way of achieving them. It leaves the determination of the method of specific improvement to the organization.
ISO 9000 is a family of quality system standards. See I. S. Organization. ISO-9000, incorporated herein in its entirety. The ISO 9000 contains a family of standards that are concerned with creating a common standard for quality management systems. ISO 9000 standards provide tools and methods to ensure that the product created by the organization regardless of its size or the complexity of the product meets the requirements defined by the customer.
ISO 9001: Quality systems—Is a standard in the ISO 9000 series of standards that can be applied to software development and maintenance. Model for quality assurance in design/development, production, installation and servicing ISO 9001 is a model used to ensure that suppliers confirm the specified requirements during design, development, production, installation and servicing. See D. Ince. (1994). ISO 9001 and software quality assurance, incorporated herein in its entirety. This model aims to achieve customer satisfaction during all stages from design to servicing, and it encourages organizations to inspect their internal quality management. In 1991 the International Standards Organization published ISO 9000-3, “Quality management and quality assurance standards—Part 3: Guidelines for the application of ISO 9001:1994 to the development, supply, installation and maintenance of computer software”. See I. S. Organization. (11-10-2014). ISO 9000-3:1991, incorporated herein in its entirety.
ISO 9001:2000 has replaced 20 points from the previous version ISO 9001:1994 standard. ISO 9001:1994 and ISO 9003:1994 quality standards are obsolete.
Bell Northen Research and Northen Telecom developed the first version of TRILLIUM in 1991 which is an assessment model designed from the customer perspective. TRILLIUM is designed for embedded telecommunication systems and includes: hardware, software, documentation, and training and support services. This model is based on the SEI's CMM that it considers the product development and support processes as a part from the organization's processes. TRILLIUM has eight capability areas, and each capability area compromises a roadmap. Each roadmap contains a set of practices that were derived from benchmark exercises.
BOOTSTRAP is a methodology for process assessment and improvement of software developed by European industry and the ESPRIT project in 1992. See P. Kuvaja, “BOOTSTRAP 3.0—A SPICE Conformant Software Process Assessment Methodology,” Software Quality Control, vol. 8, pp. 7-19, 1999, incorporated herein in its entirety. This methodology compatible with ISO/IEC 15504 contains three main phases which are: preparation, execution, and improvement planning phases. BOOTSTRAP contains five capability levels to assess an organization's processes capability for achieving their defined goals.
Al-Qutaish et al proposed the first product maturity model for assessing the quality of the software product. The proposed model was based on three models: ISO 9126, Six Sigma, and ISO 15026. The first step in determining the software product quality maturity level is to calculate the quality level using the characteristics, sub-characteristics, and measurements defined in ISO 9126, then combine the values into a single value of the quality level then covert the resulting value to six sigma. After that, find the integrity level of the software product using ISO 15026. Finally, the maturity level of the software product can be identified using FIG. 12. PMMI differs from SPQMM in having its own set of quality attributes that are collected from well-known quality models and having its own metrics that are collected from the literature which are easily applied while SPQMM is based on ISO/IEC 9126 standard's quality attributes and metrics. The limitation in SPQMM is that the assessors are forced to use the ISO 9126 quality attributes and metrics only.
The EuroScope consortium propose a maturity model of software products evaluation which is called SCOPE Maturity Model (SMM). The model has five maturity levels which are sorted from the lowest level to the highest level: initial, repeatable, defined, managed, and optimizing. The SMM model was inspired by the Capability Maturity Model (CMM). SMM levels 2, 3, and 4 use ISO 12119, ISO/IEC 9126, and ISO 14598 standards to achieve the evaluation of these levels. SMM does not focus in the final product quality (code) like PMMI. SMM is a measure of the quality in terms of matching stated specifications or requirements, and tests are executed to assess the degree to which a product meets the required specifications. SMM requires the process to be documented to ensure a product matches the specifications.
April et al. proposed the Software Maintenance Maturity Model (SMmm) that is a complement to the CMMI model. See A. April, J. Huffman Hayes, A. Abran, and R. Dumke, “Software Maintenance Maturity Model (SMmm): the software maintenance process model,” Journal of Software Maintenance and Evolution: Research and Practice, vol. 17, pp. 197-223, 2005, incorporated herein in its entirety. SMmm addresses the unique activities that are not addressed in CMMI or other models. There are five maturity levels shown in Table 7. The steps of building this model is the same steps of building Trillium. SMmm focus only on maintainability but PMMI focus on different product quality attributes including maintainability. Also SMmm does not measure the product maturity level.
TABLE 7SMmm Maturity LevelsLevelLevel nameRiskInterpretation1PerformedVery HighAd hoc maintenance process2ManagedHighBasic request-based process3EstablishedMediumState-of-the-art maintenance process4PredictableLowGenerally difficult to achieve now5OptimizingVery LowTechnologically challenging to attain
Alvaro et al proposed a Software Component Maturity Model (SCMM). The model is based on ISO/IEC9126 and ISO/IEC 14598 standards. SCMM contains five levels, the lowest level is SCMM-I and the highest is SCMM-V. See A. Alvaro, E. S. de Almeida, and S. L. Meira, “A Software Component Maturity Model (SCMM),” in Software Engineering and Advanced Applications, 2007. 33rd EUROMICRO Conference on, 2007, pp. 83-92, incorporated herein in its entirety. SCMM depends mainly on the CQM (component quality model) model which has seven characteristics where each characteristic compromises a set of life-cycle and run time characteristics as show in Table 8. Each sub-characteristic has a set of attributes that represent the entity and a metric to measure these attributes as shown in Table 9. SCMM measures only the maturity of the components and it cannot assess different types of product such as enterprise applications, web-services . . . etc.
TABLE 8A Software Component Quality model, With the Sub-CharacteristicsSub-CharacteristicsSub-CharacteristicsCharacteristics(Runtime)(Life cycle)FunctionalityAccuracySuitabilitySecurityInteroperabilityComplianceSelf-containedReliabilityFault ToleranceMaturityRecoverabilityUsabilityConfigurabilityUnderstandabilityLearnabilityOperabilityEfficiencyTime BehaviorResource BehaviorScalabilityMaintainabilityStabilityChangeabilityTestabilityPortabilityDeployabilityReplaceabilityAdaptabilityReusabilityMarketabilityDevelopment timeCostTime to marketTargeted marketAffordability
TABLE 2.9Sample Of Component Quality Attributes ForRuntime Sub-CharactcristicsSub-Character-Kindisticsof(Runtime)AttributesMetricsMetricsAccuracy1. CorrectnessTest results/precisionRSecurity2. Data EncryptionMechanism implementedP3. ControllabilityN. of interfaces/kind ofRcontrollability4. AuditabilityMechanism implementedPRecover-5. Error HandlingMechanism implementedPabilityFault6. MechanismMechanism identificationPToleranceavailable7. MechanismAmmount of errorsRefficiencytolerate/total errorsfoundConfigur-8. Effort for configureTime spend to configureIVabilitycorrectly
Golden et al. proposed the Open Source Maturity Model (OSMM). See B. Golden. (2005). Succeeding with open source, incorporated herein in its entirety. OSMM which helps TT organizations assess and make comparisons between open source software products to identify which one is the best for a defined application. This assessment method requires three phases:                1. Assess element's maturity (define requirements, locate resources, assess element maturity, and assign element score).        2. Assign for each element weighting factors.        3. Calculate overall product maturity score.        
Unfortunately, OSMM evaluates the maturity of open source products only without assessing the quality of these software products. OSMM is not primarily used to assess software product quality attributes or product maturity but to help organizations perform a comparison between open source systems.
Certifying software will increase confidence in software that passes certification. It will also make it easier to sell or purchase because there will be a reduced need for testing prior to purchase. Software certification can be granted for different types of software such as final software products and components. See P. Heck, M. Klabbers, and M. van Eekelen, “A software product certification model,” Software Quality Journal, vol. 18, pp. 37-55, Mar. 1, 2010; P. M. Heck, “A Software Product Certification Model for Dependable Systems” Eindhoven: Technische Universiteit Eindhoven2006, incorporated herein in its entirety. Certification can be provided by independent agencies which function like other quality agencies such as: Software Engineering Institute which appraises CMMI Class A or ISO which grants ISO certification. Involving external agencies in providing the certificate will increase trust in the certification as Voas says that “completely independent product certification offers the only approach that consumers can trust”. See J. Voas, “Developing a usage-based software certification process,” Computer, vol. 33, pp. 32-37, 2000; J. Morris, G. Lee, K. Parker, G. A. Bundell, and C. P. Lam, “Software component certification,” Computer, vol. 34, pp. 30-36, 2001, incorporated herein in its entirety. Most of the certification methods are process-based, from the process they can determine the quality of the final product. However, certifying the software development process only does not guarantee the quality of the final product.
Heck et al used the concept shown in FIG. 13 to propose a Software Product Certification Model for Dependable Systems. The proposed model consists of five certification levels and six product areas represented in FIG. 14 with their interrelation. Each product area is comprised of a set of elements and should achieve four generic goals (complete, uniform, correct, and consistent) as shown in Table 10. The generic goals contain a set of specific goals and properties that must be achieved to satisfy a certain level. Heck's model depends on the development process in its evaluation.
TABLE 10Generic Goals and Generic properties of the Achievement LevelGG1Complete0Some required elements are missing1All required elements are present2Semi-formal elements have been added3Formal elements have been addedGG2Uniform0No standardization1Within the product2Style complies to a company standard3Style complies to an industry standardGG3Correct (within elements)0Faults are detected1Manual review/testing has not detected any faults2Automated testing has not detected any faults3Formal verification has not detected any faultsGG4(Consistent (between elements)0Faults are detected1Manual review/testing has not detected any faults2Automated testing has not detected any faults3Formal verification has not detected any faults
Alvaro et al. propose a software component certification framework to evaluate the quality of components. The proposed framework depends on ISO/IEC 9126 and ISO/IEC 14598 and mainly on two quality models CQM and the SQuaRE (Software product quality requirements and evaluation) project. There are four steps to evaluate a component shown in FIG. 15. In the metrics framework the author uses the GQM approach to measure a component's quality attributes. Alavaro's certification model measures the components only and cannot be applied on different types of products.
Heck et al. proposed a software product certification model which is called LaQuSo (Laboratory for Quality Software) Software Product Certification Model (LSPCM) that is based on CMMI. See P. M. Heck, “A Software Product Certification Model for Dependable Systems” Eindhoven: Technische Universiteit Eindhoven2006, incorporated herein in its entirety. The proposed model has five certification levels that represent the maturity of the software products. LSPCM consist of six product areas which are the main deliverables of the development phase and each area is split into smaller parts which are called elements. The proposed model contains three certification criteria: completeness, uniformity, and conformance; and each of the certification criteria contain 4 levels as shown in Table 11. The model contains specific criteria that help achieve specific levels of certification criteria. The certification level of the final product can be computed through calculation of the certification criteria of each product area, then taking the minimum of these calculations. LSPCM's certification levels are shown in Table 12
TABLE 11Certification Criteria Achievement LevelCC1Completeness0Some required elements are missing1All required elements are present2Semiformal elements have been added3Formal elements have been addedCC2Uniformity0No standardization1Within the product2Style complies to a company standard3Style complies to an industry standardCC3Conformance0Faults are detected1Manual review/testing has not detected any faults2Automated testing has not detected any faults3Formal verification has not detected any faults
TABLE 12LSPCM's Certification levels1. InitialCC1 ≧ 1 and CC2 ≧ land CC3 =0 Each of the required elements is present in the product, and the elements are uniform. This is the level that indicates that the required elements for certification are there, and analysis can start.2. Manually verifiedCC1 ≧ 1 and CC2 ≧ 1 and CC3 = 1 All elements, relationships, and properties have been manually verified.3. Automated verifiedCC1 ≧ 2 and CC2 ≧ 2 and CC3 = 2 All elements, relationships. and properties have been verified with tool support.4. Model verifiedCC1 = 3 and CC2 = 3 and CC3 = 3 All elements, relationships and properties have been verified with mathematically-based methods wherever passible, or the most rigorous method otherwise.5. Formally verifiedCC1 = 3 and CC2 = 3 and CC3 = 3 and ‘Model Input’ Model verified where it is proven that the results of the mathematically- based methods are true for the actual input (and not only for an abstract model of it).
Heck's model evaluates the software quality based on specific criteria which do not represent all the software quality attributes; also it depends on the development process in its evaluation.
Correia et. al proposed a technical quality certification for software products that is based mainly on TSO-9126. See J. P. Correia and J. Visser, “Certification of technical quality of software products,” in Proc. of the Int'l Workshop on Foundations and Techniques for Open Source Software Certification, 2008, pp. 35-51, incorporated herein in its entirety. The authors focus on technical quality instead of functional requirements due to limitations in this approach. The focus of the study was on “Maintainability” but it can be applied for other quality attributes such as reliability. The certification is based on the relationship between the system properties and the ISO-9126 standard which can be shown in FIG. 16. The raw value of the system properties are collected using metrics. After that these attributes are mapped to the sub-characteristics of maintainability as shown in FIG. 17.
In order to find a single value of maintainability, all the calculated values of sub-characteristics' that are found from the metrics are aggregated in one value that represents the characteristic which is maintainability.
Baggen et. al proposed a maintainability certificate for software products. The approach uses the maintainability definition in ISO 9126 standard. See R. Baggen, J. P. Correia, K. Schill, and J. Visser, “Standardized code quality benchmarking for improving software maintainability,” Software Quality Journal, vol. 20, pp. 287-307, 2012; I. O. f. S. I. I. 9126-1, “Software engineering—product quality—part 1: Quality model,” incorporated herein in its entirety. The Software Improvement Group (SIG) group identifies 6 system properties which are:                1. Volume: The size of the software.        2. Redundancy: The same code can be found in different places.        3. Unit size: The units should be small and responsible for a low number of functions, in other words the unit must be cohesive.        4. Complexity: The simplicity of the code        5. Unit interface size: The number of parameters that is needed for a particular interface.        6. Coupling: The coupling between components        
The above system properties are mapped with 4 sub-characteristics of the maintainability characteristic based on the impact of each system property on the maintainability's sub-characteristics which is decided by other studies and expert opinion as shown in FIG. 18. See J. P. Correia, Y. Kanellopoulos, and J. Visser, “A survey-based study of the mapping of system properties to iso/iec 9126 maintainability characteristics,” in Software Maintenance, 2009. TCSM 2009. IEEE International Conference on, 2009, pp. 61-70, incorporated herein in its entirety.
After measuring each system property using metrics, the values of each quality attribute will be rated (each property has a separate rating table). After that the rated values will be aggregated to find a single maintainability value of the software, and certificate will be issued from TUV Informationstechnik GmbH (T″UViT) based on that value. The rating tables of the system properties are constructed using benchmark data (e.g. rating of duplication system property shown in Table 13). The quality level of the benchmark was based on the value of the calculated metrics, expert opinion, and consultants that evaluate the systems. The difference of this study from Correia et. al is the extra system properties (Unit Interface Sizing and Coupling) that are used in rating the software. See J. P. Correia and J. Visser, “Certification of technical quality of software products,” in Proc. of the Int'l Workshop on Foundations and Techniques for Open Source Software Certification, 2008, pp. 35-51, incorporated herein in its entirety.
TABLE 13Rating Table for Duplication Propertyratingduplication★★★★★ 3%★★★★ 5%★★★10%★★20%★—
Correia and Baggen certification models do not evaluate the maturity of the software product. They measure only a certain quality attribute each time. Also, they cannot measure the maturity of software product.
Yahaya et. al proposed a software product certification model. The assessment is conducted by a pre-defined interviewee answering questions (metric). See J. H. Yahaya, A. Deraman, and A. R. Hamdan, “SCfM_PROD: A software product certification model,” in Information and Communication Technologies: From Theory to Applications, 2008. ICTTA 2008. 3rd International Conference on, 2008, pp. 1-6, incorporated herein in its entirety. The model uses the Likert scale (from 1 to 5), each answer on the questionnaire has a certain point on the Likert scale so at the end the interviewee can find a single value for the attribute. A weight has been assigned to each attribute, so the certification level is calculated by summing the values found for all attributes with each multiplied by its assigned weight. The certification level of this model is shown in FIG. 19. Yahaya's model depends on the requirements phase and the metrics that are used in the model are relevant to the requirements too.
Therefore, it is established that there is no maturity model that measures the quality of the final product; most of the models in the literature only focus on the quality of the development processes as does CMMI rather than the quality of the final product based on the following premise “the quality of a system or product is highly influenced by the quality of the process used to develop and maintain it”. This work will fill this gap by developing a maturity model that will measure the quality of the final product. The proposed Product Maturity Model Integration (PMMI), will measure the different software quality attributes of the final product. Also PMMI looks at the quality of the software product not the quality of the software development processes like CMMI because the quality of the processes does not guarantee the quality of the final product. There has not been much work done on this area of software product assessment. Moreover PMMI is not restricted to certain software quality models or a set of metrics. PMMI gives flexibility to assessors to choose quality attributes that the stakeholders are interested in measuring. Furthermore PMMI can be applied to any software regardless of size and type which are developed by organizations of any size and using any software development methodologies. As a result, PMMI is a more flexible framework for the measurement of software product quality.
In view of the forgoing, the objective of the present disclosure is a framework to measure the quality of a product from multiple perspectives. The present disclosure proposes a method, apparatus, and non-transitory computer-readable storage medium that may help to objectively assess the quality of software products.